Following numato's new guide for PCIe DMA:
https://numato.com/kb/create-pcie-dma-example-design-for-nereid-2/
The relevant changes are in the constraints file in step 16.
I can actually see the device pop up with lspci:
[root@fe01 pcimem]# lspci -vv | grep -A 35 "Xilinx"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Serial controller: Xilinx Corporation Device 7024 (prog-if 01 [16450])
Subsystem: Xilinx Corporation Device 0007
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 37
Region 0: Memory at f5f00000 (32-bit, non-prefetchable) [size=1M]
Region 1: Memory at f5ef0000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 4083
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: xdma
Kernel modules: xdma
06:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)
[root@fe01 pcimem]#[root@fe01 pcimem]# lspci -vv | grep -A 35 "Xilinx"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Serial controller: Xilinx Corporation Device 7024 (prog-if 01 [16450])
Subsystem: Xilinx Corporation Device 0007
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 37
Region 0: Memory at f5f00000 (32-bit, non-prefetchable) [size=1M]
Region 1: Memory at f5ef0000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 4083
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: xdma
Kernel modules: xdma
06:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)
[root@fe01 pcimem]#For some reason I can't get the readback to work like in the guide, however:
0xf5f00000 = 4126146560
[root@fe01 pcimem]# ./pcimem /dev/mem 4126146560 w 0xffffff12 /dev/mem opened. Target offset is 0xf5f00000, page size is 4096 mmap(0, 4096, 0x3, 0x1, 3, 0xf5f00000) PCI Memory mapped to address 0x7fdef3330000. Value at offset 0xF5F00000 (0x7fdef3330000): 0xFFFFFFFF Written 0xFFFFFF12; readback 0xFFFFFFFF [root@fe01 pcimem]#
[root@fe01 pcimem]# ./pcimem /dev/mem 4126146560 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f00000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f00000)
PCI Memory mapped to address 0x7fdef3330000.
Value at offset 0xF5F00000 (0x7fdef3330000): 0xFFFFFFFF
Written 0xFFFFFF12; readback 0xFFFFFFFF
[root@fe01 pcimem]#However, this now works:
[root@fe01 tests]# ./load_driver.sh interrupt_selection . xdma 87724 0 Loading driver...insmod xdma.ko interrupt_mode=2 ... The Kernel module installed correctly and the xmda devices were recognized. DONE [root@fe01 tests]# pwd /root/dma_ip_drivers/XDMA/linux-kernel/tests [root@fe01 tests]#
[root@fe01 tests]# ./load_driver.sh
interrupt_selection .
xdma 87724 0
Loading driver...insmod xdma.ko interrupt_mode=2 ...
The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@fe01 tests]# pwd
/root/dma_ip_drivers/XDMA/linux-kernel/tests
[root@fe01 tests]#Somehow trying run_tests.sh caused the system to reboot.
[root@fe01 tests]# pwd
/root/dma_ip_drivers/XDMA/linux-kernel/tests
[root@fe01 tests]# ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
transfer size: 1024, count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0 ** Average BW = 1024, 0.003076
/dev/xdma0_h2c_1, write 0x400 @ 0x400 failed -1.
write file: Unknown error 512
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
client_loop: send disconnect: Connection reset[root@fe01 tests]# pwd
/root/dma_ip_drivers/XDMA/linux-kernel/tests
[root@fe01 tests]# ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
transfer size: 1024, count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0 ** Average BW = 1024, 0.003076
/dev/xdma0_h2c_1, write 0x400 @ 0x400 failed -1.
write file: Unknown error 512
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
client_loop: send disconnect: Connection resetI did not try following the exact steps the guide uses to communicate with dma, as they use a different file I have to move onto 'fe01'.
I also haven't tried putting the card in the linux mint machine, which may be more promising (though we had trouble with it in the past).
Following the second part of the guide (communicating using DMA) seems to work though. I had to clone this repository:
https://github.com/Kishore-Numato/XilinxAR65444/blob/master/Linux/build-install-driver-linux.sh
Then run the script build-install-driver-linux.sh in {...}/XilinxAR65444/Linux
After that I was able to follow all the steps in the guide:
[root@fe01 tests]# sudo cp ../etc/udev/rules.d/* /etc/udev/rules.d/
[root@fe01 tests]# sudo ./load_driver.sh
xdma 47131 0
Loading driver...
The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@fe01 tests]# sudo ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
transfer size: 1024
transfer count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x23e3000
CLOCK_MONOTONIC reports 0.000101002 seconds (total) for last transfer of 1024 bytes
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x24e9000
CLOCK_MONOTONIC reports 0.000033827 seconds (total) for last transfer of 1024 bytes
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0xa82000
CLOCK_MONOTONIC reports 0.000077622 seconds (total) for last transfer of 1024 bytes
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0xc16000
CLOCK_MONOTONIC reports 0.000020724 seconds (total) for last transfer of 1024 bytes
Info: Reading from c2h channel 0 at address offset 0.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
Info: Reading from c2h channel 1 at address offset 1024.
device = /dev/xdma0_c2h_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x1d61000
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x20ac000
CLOCK_MONOTONIC reports 0.000027775 seconds (total) for last transfer of 1024 bytes
CLOCK_MONOTONIC reports 0.000122408 seconds (total) for last transfer of 1024 bytes
Info: Reading from c2h channel 0 at address offset 2048.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
Info: Reading from c2h channel 1 at address offset 3072.
host memory buffer = 0x217a000
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x1eee000
CLOCK_MONOTONIC reports 0.000027601 seconds (total) for last transfer of 1024 bytes
CLOCK_MONOTONIC reports 0.000109510 seconds (total) for last transfer of 1024 bytes
Info: Checking data integrity.
Info: Data check passed for address range 0 - 1024.
Info: Data check passed for address range 1024 - 2048.
Info: Data check passed for address range 2048 - 3072.
Info: Data check passed for address range 3072 - 4096.
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.
[root@fe01 tests]#[root@fe01 tests]# sudo cp ../etc/udev/rules.d/* /etc/udev/rules.d/
[root@fe01 tests]# sudo ./load_driver.sh
xdma 47131 0
Loading driver...
The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@fe01 tests]# sudo ./run_test.sh
Info: Number of enabled h2c channels = 2
Info: Number of enabled c2h channels = 2
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
transfer size: 1024
transfer count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x23e3000
CLOCK_MONOTONIC reports 0.000101002 seconds (total) for last transfer of 1024 bytes
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x24e9000
CLOCK_MONOTONIC reports 0.000033827 seconds (total) for last transfer of 1024 bytes
Info: Writing to h2c channel 0 at address offset 2048.
Info: Writing to h2c channel 1 at address offset 3072.
Info: Wait for current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0xa82000
CLOCK_MONOTONIC reports 0.000077622 seconds (total) for last transfer of 1024 bytes
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_h2c_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0xc16000
CLOCK_MONOTONIC reports 0.000020724 seconds (total) for last transfer of 1024 bytes
Info: Reading from c2h channel 0 at address offset 0.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000000
sscanf() = 1, value = 0x00000001
Info: Reading from c2h channel 1 at address offset 1024.
device = /dev/xdma0_c2h_0, address = 0x00000000, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x1d61000
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000400, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x20ac000
CLOCK_MONOTONIC reports 0.000027775 seconds (total) for last transfer of 1024 bytes
CLOCK_MONOTONIC reports 0.000122408 seconds (total) for last transfer of 1024 bytes
Info: Reading from c2h channel 0 at address offset 2048.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000800
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_0, address = 0x00000800, size = 0x00000400, offset = 0x00000000, count = 1
Info: Reading from c2h channel 1 at address offset 3072.
host memory buffer = 0x217a000
Info: Wait for the current transactions to complete.
sscanf() = 1, value = 0x00000400
sscanf() = 1, value = 0x00000c00
sscanf() = 1, value = 0x00000001
device = /dev/xdma0_c2h_1, address = 0x00000c00, size = 0x00000400, offset = 0x00000000, count = 1
host memory buffer = 0x1eee000
CLOCK_MONOTONIC reports 0.000027601 seconds (total) for last transfer of 1024 bytes
CLOCK_MONOTONIC reports 0.000109510 seconds (total) for last transfer of 1024 bytes
Info: Checking data integrity.
Info: Data check passed for address range 0 - 1024.
Info: Data check passed for address range 1024 - 2048.
Info: Data check passed for address range 2048 - 3072.
Info: Data check passed for address range 3072 - 4096.
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.
[root@fe01 tests]#